Push model for intermediate query results

ABSTRACT

A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/995,599 entitled “Query Plans for Analytic SQL Constructs,” filed onAug. 17, 2020, which is a continuation of U.S. patent application Ser.No. 16/911,185 entitled “Query Plans for Analytic SQL Constructs,” filedon Jun. 24, 2020, now issued as U.S. Pat. No. 10,776,391 on Sep. 15,2020, which is a continuation of U.S. patent application Ser. No.16/741,676 entitled “Query Plans for Analytic SQL Constructs,” filed onJan. 13, 2020, now issued as U.S. Pat. No. 10,762,106 on Sep. 1, 2020,which is a continuation of U.S. patent application Ser. No. 14/626,853,entitled “Query Plans for Analytic SQL Constructs,” filed on Feb. 19,2015, now issued as U.S. Pat. No. 10,534,792 on Jan. 14, 2020, whichclaims the benefit of U.S. Provisional Application Ser. No. 61/941,986,entitled “Apparatus and method for enterprise data warehouse dataprocessing on cloud infrastructure,” filed Feb. 19, 2014, the disclosureof which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to resource management systems andmethods that manage data storage and computing resources.

BACKGROUND

Many existing data storage and retrieval systems are available today.For example, in a shared-disk system, all data is stored on a sharedstorage device that is accessible from all of the processing nodes in adata cluster. In this type of system, all data changes are written tothe shared storage device to ensure that all processing nodes in thedata cluster access a consistent version of the data. As the number ofprocessing nodes increases in a shared-disk system, the shared storagedevice (and the communication links between the processing nodes and theshared storage device) becomes a bottleneck that slows data read anddata write operations. This bottleneck is further aggravated with theaddition of more processing nodes. Thus, existing shared-disk systemshave limited scalability due to this bottleneck problem.

Another existing data storage and retrieval system is referred to as a“shared-nothing architecture.” In this architecture, data is distributedacross multiple processing nodes such that each node stores a subset ofthe data in the entire database. When a new processing node is added orremoved, the shared-nothing architecture must rearrange data across themultiple processing nodes. This rearrangement of data can betime-consuming and disruptive to data read and write operations executedduring the data rearrangement. Further, this architecture requires atleast one processing node to store data. Thus, the shared-nothingarchitecture fails to store data if all processing nodes are removed.Additionally, management of data in a shared-nothing architecture iscomplex due to the distribution of data across many different processingnodes.

The systems and methods described herein provide an improved approach todata storage and data retrieval that alleviates the above-identifiedlimitations of existing systems.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present disclosureare described with reference to the following figures, wherein likereference numerals refer to like parts throughout the various figuresunless otherwise specified.

FIG. 1A illustrates an information flow diagram depicting a join processwithin a distributed system.

FIG. 1B illustrates an information flow diagram depicting a join processwithin a distributed system.

FIG. 1C illustrates an information flow diagram depicting a join processwithin a distributed system.

FIG. 2 is a process flow diagram depicting an implementation of themethods disclosed herein.

FIG. 3 illustrates a block diagram depicting an embodiment of anoperating environment in accordance with the teachings of thedisclosure.

FIG. 4 illustrates a block diagram depicting an example of animplementation of a resource manager in accordance with the teachings ofthe disclosure.

FIG. 5 illustrates a block diagram depicting an example of animplementation of a execution platform in accordance with the teachingsof the disclosure.

FIG. 6 illustrates a block diagram depicting an example computing devicein accordance with the teachings of the disclosure.

DETAILED DESCRIPTION

Disclosed herein are systems, apparatuses and methods for managing datastorage and data access for querying data in a distributed systemwithout buffering intermediate results to disk storage during use. Forexample, an implementation may comprise an access module configured foraccessing a query plan having a plurality of operators, wherein a firstoperator in the query plan processes a set of data to generate a firstresult. Additionally, after first operator has begun to produce anoutput the output is pushed to a plurality of secondary operators togenerate a second result from the output. In the disclosed systems,because a subsequent operator is enabled by the disclosed methods tobegin working on any amount of output as it is produced from a precedingoperator, the buffering needed between operators may be greatly reducedor eliminated.

In the following description, reference is made to the accompanyingdrawings that form a part thereof, and in which is shown by way ofillustration specific exemplary embodiments in which the disclosure maybe practiced. These embodiments are described in sufficient detail toenable those skilled in the art to practice the concepts disclosedherein, and it is to be understood that modifications to the variousdisclosed embodiments may be made, and other embodiments may beutilized, without departing from the scope of the present disclosure.The following detailed description is, therefore, not to be taken in alimiting sense.

Reference throughout this specification to “one embodiment,” “anembodiment,” “one example” or “an example” means that a particularfeature, structure or characteristic described in connection with theembodiment or example is included in at least one embodiment of thepresent disclosure. Thus, appearances of the phrases “in oneembodiment,” “in an embodiment,” “one example” or “an example” invarious places throughout this specification are not necessarily allreferring to the same embodiment or example. Furthermore, the particularfeatures, structures, databases or characteristics may be combined inany suitable combinations and/or sub-combinations in one or moreembodiments or examples. In addition, it should be appreciated that thefigures provided herewith are for explanation purposes to personsordinarily skilled in the art and that the drawings are not necessarilydrawn to scale.

Embodiments in accordance with the present disclosure may be embodied asan apparatus, method or computer program product. Accordingly, thepresent disclosure may take the form of an entirely hardware-comprisedembodiment, an entirely software-comprised embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,embodiments of the present disclosure may take the form of a computerprogram product embodied in any tangible medium of expression havingcomputer-usable program code embodied in the medium.

Any combination of one or more computer-usable or computer-readablemedia may be utilized. For example, a computer-readable medium mayinclude one or more of a portable computer diskette, a hard disk, arandom access memory (RAM) device, a read-only memory (ROM) device, anerasable programmable read-only memory (EPROM or Flash memory) device, aportable compact disc read-only memory (CDROM), an optical storagedevice, and a magnetic storage device. Computer program code forcarrying out operations of the present disclosure may be written in anycombination of one or more programming languages. Such code may becompiled from source code to computer-readable assembly language ormachine code suitable for the device or computer on which the code willbe executed.

Embodiments may also be implemented in cloud computing environments. Inthis description and the following claims, “cloud computing” may bedefined as a model for enabling ubiquitous, convenient, on-demandnetwork access to a shared pool of configurable computing resources(e.g., networks, servers, storage, applications, and services) that canbe rapidly provisioned via virtualization and released with minimalmanagement effort or service provider interaction and then scaledaccordingly. A cloud model can be composed of various characteristics(e.g., on-demand self-service, broad network access, resource pooling,rapid elasticity, and measured service), service models (e.g., Softwareas a Service (“SaaS”), Platform as a Service (“PaaS”), andInfrastructure as a Service (“IaaS”)), and deployment models (e.g.,private cloud, community cloud, public cloud, and hybrid cloud).

The flow diagrams and block diagrams in the attached figures illustratethe architecture, functionality, and operation of possibleimplementations of systems, methods, and computer program productsaccording to various embodiments of the present disclosure. In thisregard, each block in the flow diagrams or block diagrams may representa module, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It will also be noted that each block of the block diagramsand/or flow diagrams, and combinations of blocks in the block diagramsand/or flow diagrams, may be implemented by special purposehardware-based systems that perform the specified functions or acts, orcombinations of special purpose hardware and computer instructions.These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flow diagram and/orblock diagram block or blocks.

The systems and methods described herein provide a flexible and scalabledata warehouse using a new data processing platform. In someembodiments, the described systems and methods leverage a cloudinfrastructure that supports cloud-based storage resources, computingresources, and the like. Example cloud-based storage resources offersignificant storage capacity available on-demand at a low cost. Further,these cloud-based storage resources may be fault-tolerant and highlyscalable, which can be costly to achieve in private data storagesystems. Example cloud-based computing resources are available on-demandand may be priced based on actual usage levels of the resources.Typically, the cloud infrastructure is dynamically deployed,reconfigured, and decommissioned in a rapid manner.

In the described systems and methods, a data storage system utilizes anSQL (Structured Query Language)-based relational database. However, thesystems and methods disclosed herein are applicable to any type ofdatabase using any data storage architecture and using any language tostore and retrieve data within the database. Additionally, the systemsand methods described herein further provide a multi-tenant system thatsupports isolation of computing resources and data between differentcustomers/clients and between different users within the samecustomer/client.

A relational join is one of the fundamental data processing operationsin a relational data management system. A join is a binary operator,taking two relations R and S, and a binary predicate θ as inputs, andproducing a single relation R[?]_(θ) S which contains the set of allcombinations of tuples in R and S which satisfy the predicate θ.

A single query typically performs multiple such join operations,resulting in a tree-shaped execution plan. Join operations formintermediate nodes and group nodes of that tree shape, while baserelations form analogous leaves of that tree. Data flows from the leavesof the tree towards the root, where the final query result is produced.The execution time of a query is directly related to the amount ofintermediate data it produces. Relational data management systems thusseek to minimize the amount of intermediate data which flows through thetree in order to minimize the execution time of the query.

This disclosure discloses a method for pushing results of primaryoperators to an operator further down the tree, without having to bufferthe results. Such methods are advantageous in SQL constructs. There areseveral SQL constructs which are frequently used in analytics queries,which rely on scanning an intermediate result many times in order tocompute a result. These may include: grouping sets, rollup, and cubeaggregates. These SQL constructs allow aggregations along multipledimensions of a data set to be generated into a single result set. OtherSQL constructs, referred to as “window aggregates” allow aggregatesacross different partitions of a dataset to be produced into a singleresult set. In another SQL construct, distinct aggregates compute anaggregate function on an expression, ignoring duplicate values of theexpression because multiple distinct aggregates require eliminatingduplicates along different attributes of the same intermediate data set.

The intermediate result that is input to these forms of aggregation canbe a base table, or the result of earlier processing, e.g. filters,joins, aggregations, etc., which could be arbitrarily complex.

Current database systems evaluate such queries in one of two ways: Theinput relation is recomputed for each step of the plan that consumes theinput. This can be undesirable if the input relation is the result ofexpensive operations. This technique is referred to as “inlining” of theinput relation. The query may also be evaluated in two steps. The firststep evaluates the input relation and materializes its result to atemporary structure that is typically stored and/or buffered in memory,or written to disk if the input relation is too large for memory. Thesecond step evaluates the aggregation, with each consumer of the inputrelation reading from the temporary structure. The resource cost ofmaterializing and reading back the input relation can be large, e.g. ifthe result does not fit in memory and must be written to disk, resultingin higher economic and operational costs. Also, materializing theintermediate result is a blocking operator, which breaks the pipeliningof the plan. As used herein the terms “cost of materializing” and “costof reading back” are intended to denote all of the resources, expenses,and costs associated with materializing and reading back results fromvarious operators, both operationally and economically. This techniquemay be referred to as “materialization” of the input relation.

An example of a grouping sets query is illustrated as Q1 in FIG. 1A:

-   -   Q1    -   select t1.a, t2.b, count(*)    -   from t1, t2    -   where t1.x=t2.y    -   group by grouping sets (t1.a, t2.b);

Using the above in described inlining, the following equivalent query(illustrated as Q2 in FIG. 1B) would be executed:

-   -   Q2    -   select t1.a, null as t2.b, count(*)    -   from t1, t2    -   where t1.x=t2.y    -   group by t1.a    -   union all    -   select null as t1.a, t2.b, count(*)    -   from t1, t2    -   where t1.x=t2.y    -   group by t2.b;

Using materialization, the following equivalent sequence of queriesillustrated in FIG. 1C would be executed:

-   -   Q3a    -   create table gsets_temp_tab as    -   select t1.a, t2.b    -   from t1, t2    -   where t1.x=t2.y;    -   select t1.a, null as t2.b, count(*)    -   from gsets_temp_tab    -   group by t1.a    -   Q3b    -   union all    -   select null as t1.a, t2.b, count(*)    -   from gsets_temp_tab    -   group by t2.b;    -   drop table gsets_temp_tab;

Most commercial database systems use a combination of the abovetechniques, using either heuristics or a cost-model to determine whethermaterialization or inlining is likely to be most efficient for aparticular aggregation in a particular query. In the example above, themore optimal execution technique would depend on how costly it is tocompute the join of t1 and t2, versus the cost of materializing theresult, which would depend on the size of the result.

An improved execution model may use a “push model” to schedule operatorsin a plan. When one operator finishes processing data, it pushes itsresults to the operator who consumes the result, and that operator thendoes its work. This model allows an operator to push its result tomultiple secondary operators/consumers of the same intermediate resultfor concurrent processing. This is advantageous for implementing SQLconstructs that require processing the same data set in different ways.As used herein the phrase “unique operators” is intended to denotefirst, second, and n-operators that are different in form or functionfrom other operators within a query plan or portion of a query plan.

Further, as illustrated in FIG. 1A, query Q1 processed herein using themethod 200 of FIG. 2, disclosed for the execution model, the result ofthe join operator 110 can be directly pushed to both of the group-bynodes 120 a, 120 b in the expanded query (Q2) without having to bufferthe results in disk storage. In this embodiment the group-by nodes 120a, 120 b may be secondary or intermediate operators that may process theresults of the join operator 110 with different operations. After all ofthe group-by nodes 120 a, 120 b have consumed the results, the resultsmay be finally aggregated by a final join operation 130.

Using this kind of plan avoids both re-computing the result of the join,and the cost of materializing the result of the join. The group-byoperators can execute concurrently, such that the pipeline from the jointo the group-bys is not broken. Additionally, in an implementation, eachof the secondary operators may be different from one another, such thatthe same intermediate data may be processed differently as may berequired in SQL constructs.

Illustrated in FIG. 2 is a flow diagram for a method 200 of performing ajoin operation without the need to buffer in disk storage. At 210 of themethod 200, a query plan having a plurality of operators may beaccessed. It should be noted that in an implementation, each of theplurality of operators may comprise differing processes for the sameresult it consumes. The plurality of operators may be run concurrentlyand timed so that their results are coordinated properly for a finaljoin operation.

At 220, the method 200 may call for the initiation of a first operatorin the query plan to process a set of data and generate a first result.At 230 it may be determined whether the first operator has completedprocessing the set of data, and responsive to determining that the firstoperator has generated some output, pushing the first operator output toa plurality of other secondary operators in the query plan at 240.

Finally, at 250, initiating each of the other secondary operators toprocess the first result to generate a second result that may result ina final join that may be stored on disk memory at 260.

In contrast, other commercial database systems use a pull model, whichforces a decision between either the inlining or materialization methodsdescribed above.

The methods disclosed herein may be used to improve performance ofanalytics queries in a distributed relation system.

Illustrated in FIG. 3 is a computer system for running the methodsdisclosed herein. As shown in FIG. 3, a resource manager 302 is coupledto multiple users 304, 306, and 308. In particular implementations,resource manager 302 can support any number of users desiring access todata processing platform 300. Users 304-308 may include, for example,end users providing data storage and retrieval requests, systemadministrators managing the systems and methods described herein, andother components/devices that interact with resource manager 302.Resource manager 302 provides various services and functions thatsupport the operation of all systems and components within dataprocessing platform 300. Resource manager 302 is also coupled tometadata 310, which is associated with the entirety of data storedthroughout data processing platform 300. In some embodiments, metadata310 includes a summary of data stored in remote data storage systems aswell as data available from a local cache. Additionally, metadata 310may include information regarding how data is organized in the remotedata storage systems and the local caches. Metadata 310 allows systemsand services to determine whether a piece of data needs to be processedwithout loading or accessing the actual data from a storage device.

Resource manager 302 is further coupled to an execution platform 312,which provides multiple computing resources that execute various datastorage and data retrieval tasks, as discussed in greater detail below.Execution platform 312 is coupled to multiple data storage devices 316,318, and 320 that are part of a storage platform 314. Although threedata storage devices 316, 318, and 320 are shown in FIG. 3, executionplatform 312 is capable of communicating with any number of data storagedevices. In some embodiments, data storage devices 316, 318, and 320 arecloud-based storage devices located in one or more geographic locations.For example, data storage devices 316, 318, and 320 may be part of apublic cloud infrastructure or a private cloud infrastructure. Datastorage devices 316, 318, and 320 may be hard disk drives (HDDs), solidstate drives (SSDs), storage clusters or any other data storagetechnology. Additionally, storage platform 314 may include distributedfile systems (such as Hadoop Distributed File Systems (HDFS)), objectstorage systems, and the like.

In particular embodiments, the communication links between resourcemanager 302 and users 304-308, metadata 310, and execution platform 312are implemented via one or more data communication networks. Similarly,the communication links between execution platform 312 and data storagedevices 316-320 in storage platform 314 are implemented via one or moredata communication networks. These data communication networks mayutilize any communication protocol and any type of communication medium.In some embodiments, the data communication networks are a combinationof two or more data communication networks (or sub-networks) coupled toone another. In alternate embodiments, these communication links areimplemented using any type of communication medium and any communicationprotocol.

As shown in FIG. 3, data storage devices 316, 318, and 320 are decoupledfrom the computing resources associated with execution platform 312.This architecture supports dynamic changes to data processing platform300 based on the changing data storage/retrieval needs as well as thechanging needs of the users and systems accessing data processingplatform 300. The support of dynamic changes allows data processingplatform 300 to scale quickly in response to changing demands on thesystems and components within data processing platform 300. Thedecoupling of the computing resources from the data storage devicessupports the storage of large amounts of data without requiring acorresponding large amount of computing resources. Similarly, thisdecoupling of resources supports a significant increase in the computingresources utilized at a particular time without requiring acorresponding increase in the available data storage resources.

Resource manager 302, metadata 310, execution platform 312, and storageplatform 314 are shown in FIG. 3 as individual components. However, eachof resource manager 302, metadata 310, execution platform 312, andstorage platform 314 may be implemented as a distributed system (e.g.,distributed across multiple systems/platforms at multiple geographiclocations). Additionally, each of resource manager 302, metadata 310,execution platform 312, and storage platform 314 can be scaled up ordown (independently of one another) depending on changes to the requestsreceived from users 304-308 and the changing needs of data processingplatform 300. Thus, in the described embodiments, data processingplatform 300 is dynamic and supports regular changes to meet the currentdata processing needs.

FIG. 4 is a block diagram depicting an embodiment of resource manager302. As shown in FIG. 3, resource manager 302 includes an access manager402 and a key manager 404 coupled to a data storage device 406. Accessmanager 402 handles authentication and authorization tasks for thesystems described herein. Key manager 404 manages storage andauthentication of keys used during authentication and authorizationtasks. A request processing service 408 manages received data storagerequests and data retrieval requests. A management console service 410supports access to various systems and processes by administrators andother system managers.

Resource manager 302 also includes an SQL compiler 412, an SQL optimizer414 and an SQL executor 410. SQL compiler 412 parses SQL queries andgenerates the execution code for the queries. SQL optimizer 414determines the best method to execute queries based on the data thatneeds to be processed. SQL executor 416 executes the query code forqueries received by resource manager 302. A query scheduler andcoordinator 418 sends received queries to the appropriate services orsystems for compilation, optimization, and dispatch to an executionplatform. A virtual warehouse manager 420 manages the operation ofmultiple virtual warehouses implemented in an execution platform.

Additionally, resource manager 302 includes a configuration and metadatamanager 422, which manages the information related to the data stored inthe remote data storage devices and in the local caches. A monitor andworkload analyzer 424 oversees the processes performed by resourcemanager 302 and manages the distribution of tasks (e.g., workload)across the virtual warehouses and execution nodes in the executionplatform. Configuration and metadata manager 422 and monitor andworkload analyzer 424 are coupled to a data storage device 426.

Resource manager 302 also includes a transaction management and accesscontrol module 428, which manages the various tasks and other activitiesassociated with the processing of data storage requests and data accessrequests. For example, transaction management and access control module428 provides consistent and synchronized access to data by multipleusers or systems. Since multiple users/systems may access the same datasimultaneously, changes to the data must be synchronized to ensure thateach user/system is working with the current version of the data.Transaction management and access control module 428 provides control ofvarious data processing activities at a single, centralized location inresource manager 302.

FIG. 5 is a block diagram depicting an embodiment of an executionplatform. As shown in FIG. 5, execution platform 512 includes multiplevirtual warehouses 502, 504, and 506. Each virtual warehouse includesmultiple execution nodes that each include a cache and a processor.Although each virtual warehouse 502-506 shown in FIG. 5 includes threeexecution nodes, a particular virtual warehouse may include any numberof execution nodes. Further, the number of execution nodes in a virtualwarehouse is dynamic, such that new execution nodes are created whenadditional demand is present, and existing execution nodes are deletedwhen they are no longer necessary.

Each virtual warehouse 502-506 is capable of accessing any of the datastorage devices 316-320 shown in FIG. 3. Thus, virtual warehouses502-506 are not necessarily assigned to a specific data storage device316-320 and, instead, can access data from any of the data storagedevices 316-320. Similarly, each of the execution nodes shown in FIG. 5can access data from any of the data storage devices 316-320. In someembodiments, a particular virtual warehouse or a particular executionnode may be temporarily assigned to a specific data storage device, butthe virtual warehouse or execution node may later access data from anyother data storage device.

In the example of FIG. 5, virtual warehouse 502 includes three executionnodes 508, 510, and 512. Execution node 508 includes a cache 514 and aprocessor 516. Execution node 510 includes a cache 518 and a processor520. Execution node 512 includes a cache 522 and a processor 524. Eachexecution node 508-512 is associated with processing one or more datastorage and/or data retrieval tasks. For example, a particular virtualwarehouse may handle data storage and data retrieval tasks associatedwith a particular user or customer. In other implementations, aparticular virtual warehouse may handle data storage and data retrievaltasks associated with a particular data storage system or a particularcategory of data.

Similar to virtual warehouse 502 discussed above, virtual warehouse 504includes three execution nodes 526, 528, and 530. Execution node 526includes a cache 532 and a processor 534. Execution node 528 includes acache 536 and a processor 538. Execution node 530 includes a cache 540and a processor 542. Additionally, virtual warehouse 506 includes threeexecution nodes 544, 546, and 548. Execution node 544 includes a cache550 and a processor 552. Execution node 546 includes a cache 554 and aprocessor 556. Execution node 548 includes a cache 558 and a processor560.

Although the execution nodes shown in FIG. 5 each include one cache andone processor, alternate embodiments may include execution nodescontaining any number of processors and any number of caches.Additionally, the caches may vary in size among the different executionnodes. The caches shown in FIG. 5 store, in the local execution node,data that was retrieved from one or more data storage devices in astorage platform 314 (FIG. 3). Thus, the caches reduce or eliminate thebottleneck problems occurring in platforms that consistently retrievedata from remote storage systems. Instead of repeatedly accessing datafrom the remote storage devices, the systems and methods describedherein access data from the caches in the execution nodes which issignificantly faster and avoids the bottleneck problem discussed above.In some embodiments, the caches are implemented using high-speed memorydevices that provide fast access to the cached data. Each cache canstore data from any of the storage devices in storage platform 314.

Further, the cache resources and computing resources may vary betweendifferent execution nodes. For example, one execution node may containsignificant computing resources and minimal cache resources, making theexecution node useful for tasks that require significant computingresources. Another execution node may contain significant cacheresources and minimal computing resources, making this execution nodeuseful for tasks that require caching of large amounts of data. In someembodiments, the cache resources and computing resources associated witha particular execution node are determined when the execution node iscreated, based on the expected tasks to be performed by the executionnode.

Additionally, the cache resources and computing resources associatedwith a particular execution node may change over time based on changingtasks performed by the execution node. For example, a particularexecution node may be assigned more processing resources if the tasksperformed by the execution node become more processor intensive.Similarly, an execution node may be assigned more cache resources if thetasks performed by the execution node require a larger cache capacity.

Although virtual warehouses 502-506 are associated with the sameexecution platform 312 of FIG. 3, the virtual warehouses may beimplemented using multiple computing systems at multiple geographiclocations. For example, virtual warehouse 502 can be implemented by acomputing system at a first geographic location, while virtualwarehouses 504 and 506 are implemented by another computing system at asecond geographic location. In some embodiments, these differentcomputing systems are cloud-based computing systems maintained by one ormore different entities.

Additionally, each virtual warehouse is shown in FIG. 5 as havingmultiple execution nodes. The multiple execution nodes associated witheach virtual warehouse may be implemented using multiple computingsystems at multiple geographic locations. For example, a particularinstance of virtual warehouse 502 implements execution nodes 508 and 510on one computing platform at a particular geographic location, andimplements execution node 512 at a different computing platform atanother geographic location. Selecting particular computing systems toimplement an execution node may depend on various factors, such as thelevel of resources needed for a particular execution node (e.g.,processing resource requirements and cache requirements), the resourcesavailable at particular computing systems, communication capabilities ofnetworks within a geographic location or between geographic locations,and which computing systems are already implementing other executionnodes in the virtual warehouse. Execution platform 312 is also faulttolerant. For example, if one virtual warehouse fails, that virtualwarehouse is quickly replaced with a different virtual warehouse at adifferent geographic location.

A particular execution platform 312 may include any number of virtualwarehouses 502-506. Additionally, the number of virtual warehouses in aparticular execution platform is dynamic, such that new virtualwarehouses are created when additional processing and/or cachingresources are needed. Similarly, existing virtual warehouses may bedeleted when the resources associated with the virtual warehouse are nolonger necessary.

FIG. 6 is a block diagram depicting an example computing device 600. Insome embodiments, computing device 600 is used to implement one or moreof the systems and components discussed herein. For example, computingdevice 600 may allow a user or administrator to access resource manager302. Further, computing device 600 may interact with any of the systemsand components described herein. Accordingly, computing device 600 maybe used to perform various procedures and tasks, such as those discussedherein. Computing device 600 can function as a server, a client or anyother computing entity. Computing device 600 can be any of a widevariety of computing devices, such as a desktop computer, a notebookcomputer, a server computer, a handheld computer, a tablet, and thelike.

Computing device 600 includes one or more processor(s) 602, one or morememory device(s) 604, one or more interface(s) 606, one or more massstorage device(s) 608, and one or more Input/Output (I/O) device(s) 610,all of which are coupled to a bus 612. Processor(s) 602 include one ormore processors or controllers that execute instructions stored inmemory device(s) 604 and/or mass storage device(s) 608. Processor(s) 602may also include various types of computer-readable media, such as cachememory.

Memory device(s) 604 include various computer-readable media, such asvolatile memory (e.g., random access memory (RAM)) and/or nonvolatilememory (e.g., read-only memory (ROM)). Memory device(s) 604 may alsoinclude rewritable ROM, such as Flash memory.

Mass storage device(s) 608 include various computer readable media, suchas magnetic tapes, magnetic disks, optical disks, solid state memory(e.g., Flash memory), and so forth. Various drives may also be includedin mass storage device(s) 608 to enable reading from and/or writing tothe various computer readable media. Mass storage device(s) 608 includeremovable media and/or non-removable media.

I/O device(s) 610 include various devices that allow data and/or otherinformation to be input to or retrieved from computing device 600.Example I/O device(s) 610 include cursor control devices, keyboards,keypads, microphones, monitors or other display devices, speakers,printers, network interface cards, modems, lenses, CCDs or other imagecapture devices, and the like.

Interface(s) 606 include various interfaces that allow computing device600 to interact with other systems, devices, or computing environments.Example interface(s) 606 include any number of different networkinterfaces, such as interfaces to local area networks (LANs), wide areanetworks (WANs), wireless networks, and the Internet.

Bus 612 allows processor(s) 602, memory device(s) 604, interface(s) 606,mass storage device(s) 608, and I/O device(s) 610 to communicate withone another, as well as other devices or components coupled to bus 612.Bus 612 represents one or more of several types of bus structures, suchas a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.

For purposes of illustration, programs and other executable programcomponents are shown herein as discrete blocks, although it isunderstood that such programs and components may reside at various timesin different storage components of computing device 600, and areexecuted by processor(s) 602. Alternatively, the systems and proceduresdescribed herein can be implemented in hardware, or a combination ofhardware, software, and/or firmware. For example, one or moreapplication specific integrated circuits (ASICs) can be programmed tocarry out one or more of the systems and procedures described herein.

Although the present disclosure is described in terms of certainpreferred embodiments, other embodiments will be apparent to those ofordinary skill in the art, given the benefit of this disclosure,including embodiments that do not provide all of the benefits andfeatures set forth herein, which are also within the scope of thisdisclosure. It is to be understood that other embodiments may beutilized, without departing from the scope of the present disclosure.

What is claimed is:
 1. A method comprising: accessing a query plancomprising a step to be performed on a set of data pertaining to aquery; initiating, within one or more of a first set of processors, afirst parallel execution process in the query plan to process the set ofdata and generate an intermediate result of the query; providing, by theone or more of the first set of processors, during execution of thequery plan, the intermediate result to a second set of processors forprocessing a plurality of secondary parallel execution processes withinthe second set of processors, wherein each of the first set ofprocessors and the second set of processors is decoupled from a datastorage device; initiating, by the one or more of the second set ofprocessors, the plurality of secondary parallel execution processes toprocess the intermediate result to generate a plurality of secondaryresults; and storing a final result to storage within a storageplatform.
 2. The method of claim 1, wherein the plurality of secondaryresults generated without writing the intermediate result to a datastorage device.
 3. The method of claim 1, wherein each of the pluralityof secondary parallel execution processes process the intermediateresult using a different operation.
 4. The method of claim 1, furthercomprising: delaying operation of at least one of the plurality ofsecondary parallel execution processes so as to coordinate timing amongother secondary parallel execution processes of the plurality ofsecondary parallel execution processes.
 5. The method of claim 1,wherein the intermediate result comprises a plurality of rows ofdatabase data.
 6. The method of claim 1, wherein each of the pluralityof secondary parallel execution processes are unique operators.
 7. Themethod of claim 1, wherein the intermediate result is not materialized.8. The method of claim 1, wherein the intermediate result generated bythe first parallel execution processes is not materialized to atemporary structure.
 9. The method of claim 1, wherein the set of datais stored in a relational database.
 10. The method of claim 9, whereinthe relational database is a structured query language database.
 11. Themethod of claim 1, wherein one or more databases storing the set of datacomprise a multi-tenant database that isolates computing resources anddata between different customers.
 12. The method of claim 1, wherein theeach of the first set of one or more processors and the second set ofone or more processors is coupled to the data storage device via acommunications network.
 13. The method of claim 1, wherein the each ofthe first set of one or more processors and the second set of one ormore processors includes a local cache.
 14. A system comprising: a querycoordinating process programmed to: access a query plan comprising astep to be performed on a set of data referenced by a query; a first setof processors programmed to: initiate a first parallel execution processin the query plan to process the set of data and generate anintermediate result of the query, and provide, during execution of thequery plan, the intermediate result to a second set of processors forconcurrent processing by the second set of processors, wherein each ofthe first set of processors and the second set of processors isdecoupled from a data storage device; and the second set of processorsprogrammed to initiate a plurality of secondary parallel executionprocesses to process the intermediate result to generate a plurality ofsecondary results; and a storage platform programmed to store a finalresult to storage.
 15. The system of claim 14, wherein the plurality ofsecondary results generated without writing the intermediate result to adata storage device.
 16. The system of claim 14, wherein each of theplurality of secondary parallel execution processes process theintermediate result using a different operation.
 17. The system of claim14, the first set of processors further programmed to: delay operationof at least one of the plurality of secondary parallel executionprocesses so as to coordinate timing among other secondary parallelexecution processes of the plurality of secondary parallel executionprocesses.
 18. The system of claim 14, wherein the intermediate resultcomprises a plurality of rows of database data.
 19. The system of claim14, wherein each of the plurality of secondary parallel executionprocesses are unique operators.
 20. The system of claim 14, wherein theintermediate result is not materialized.
 21. The system of claim 14,wherein the intermediate result generated by the first parallelexecution processes is not materialized to a temporary structure. 22.The system of claim 14, wherein the set of data is stored in arelational database.
 23. The system of claim 22, wherein the relationaldatabase is a structured query language database.
 24. The system ofclaim 14, wherein one or more databases storing the data set comprise amulti-tenant database that isolates computing resources and data betweendifferent customers.
 25. The system of claim 14, wherein the each of thefirst set of one or more processors and the second set of one or moreprocessors is coupled to the data storage device via a communicationsnetwork.
 26. The system of claim 14, wherein the each of the first setof one or more processors and the second set of one or more processorsincludes a local cache.
 27. A non-transitory computer-readable mediumstoring instructions which, when executed by one or more third set ofone or more processors of a computing system, cause the computing systemto: access a query plan comprising a step to be performed on a set ofdata referenced by a query; initiate, within a first set of one or moreprocessors, a first parallel execution process in the query plan toprocess the set of data and generate an intermediate result of thequery; provide, by the first processor, during execution of the queryplan, the intermediate result to a second set of processors forprocessing a plurality of secondary parallel execution processes by thesecond set of processors, wherein each of the first set and the secondset of processors are decoupled from a data storage device; initiate, byone or more of the second set of processors, the plurality of secondaryparallel execution processes to process the intermediate result togenerate a plurality of secondary results; and store a final result tostorage within a storage platform.
 28. The non-transitorycomputer-readable medium of claim 27, wherein the plurality of secondaryresults is generated without writing the intermediate result to the datastorage device.
 29. The non-transitory computer-readable medium of claim27, wherein the query coordinating process further to: delay operationof at least one of the plurality of secondary parallel executionprocesses so as to coordinate timing among other secondary parallelexecution processes of the plurality of secondary parallel executionprocesses.
 30. The non-transitory computer-readable medium of claim 27,wherein the each of the first set of one or more processors and thesecond set of one or more processors is coupled to the data storagedevice via a communications network.